Real-time communications architecture and methods for use with a personal computer system

ABSTRACT

A system and method for improved real-time communications in a computer system that eliminates the distinctions between various communication devices, communication channels and types of communication, including voice, text and video, is provided. When connected for communication, the user may easily move between modalities, including switching devices on which the user is communicating, switching networks, and transitioning between types of communication. A computer system includes various integrated communications-related devices, and a controller controls those devices based on user actions entered via buttons or user interface software and provides notifications to the user. The user is able to input data via any appropriate input mechanism, and receive data on any suitable output mechanism. Improved non-linear caller ID navigation is also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.11/837,302, filed Aug. 10, 2007, and entitled Real-Time CommunicationsArchitecture and Methods for Use With a Personal Computer System, whichis a divisional application of U.S. patent application Ser. No.10/429,905 filed May 5, 2003, and entitled Real-Time CommunicationsArchitecture And Methods For Use With A Personal Computer System whichis incorporated herein by reference in its entirety. That application isrelated to the following copending United States patent applications,assigned to the assignee of the present invention, and herebyincorporated by reference in their entireties:

U.S. patent application Ser. No. 10/429,932 filed May 5, 2003, andentitled “Method and System for Auxiliary Display of Information for aComputing Device,”;

U.S. patent application Ser. No. 10/429,904 filed May 5, 2003, andentitled “Record Button on a Computer System,”;

U.S. patent application Ser. No. 10/429,930 filed May 5, 2003, andentitled “Method and System for Auxiliary Processing Of Information fora Computing Device,”;

U.S. patent application Ser. No. 10/430,369 filed May 5, 2003, andentitled “System and Method for Activating a Computer System,”;

U.S. patent application Ser. No. 10/429,903 filed May 5, 2003, andentitled “Computer System with Do Not Disturb System and Method,”;

U.S. patent application Ser. No. 10/429,943 filed May 5, 2003, andentitled “Computer Camera System and Method for Reducing Parallax,”;

U.S. patent application Ser. No. 10/429,933 filed May 5, 2003, andentitled “Control and Communications Panel for a Computer System,”; and

U.S. patent application Ser. No. 10/429,931 filed May 5, 2003, andentitled “Notification Lights, Locations and Rules for a ComputerSystem.”.

FIELD OF THE INVENTION

The invention relates generally to communications and computer systems.

BACKGROUND OF THE INVENTION

People communicate in many ways over many types of devices and channels,including by voice, by audio/video, and by text or the like. Voicecommunications include audio over landline telephones such as connectedto POTS (plain old telephone service) lines or PBX (private branchexchange) lines, and cellular and satellite telephones, as well as voiceover Internet Protocol (VoIP). Audiovisual communications include videotele-conference (VTC), which also may be Internet Protocol-basedcommunication. Text communications include electronic mail, instantmessaging, pagers with text messaging, and so forth.

Some contemporary devices provide multiple ways to communicate, e.g.,pocket sized personal computers may have a cellular telephone connectionto place audio calls, and a wireless internet connection for sending andreceiving electronic mail messages and instant messaging communications.Some mobile devices (e.g., a Smartphone) are considered to be more liketelephones, but likewise provide Internet access, and may handle textmessaging generally.

While all of these technologies are fairly easy to use, from the user'sperspective there is still a substantial emphasis on the communicationchannels and protocols, and the devices themselves, rather than on whatis really important to the user, which is communicating with someoneelse over an available device. For example, a user who needs to talk tosomeone else may need to consider many factors before selecting a deviceand attempting to call the other person, such as whether to use alandline phone or a mobile telephone, e.g., because the call may take along time, part of which could be spent while traveling. Other factorsthat a user may consider include the other device that the caller shouldbe calling (whether to call the recipient's landline telephone or mobiletelephone), the cost of the call, whether the call should be a videocall, and so forth. Each of the devices in turn has its ownidiosyncrasies, such as a custom user interface, a different set ofstored numbers from those stored in other devices, and so forth.

What is needed is an automated architecture (system) and related methodsfor hiding much of the device and connection considerations from users,and replacing those considerations with a model that focuses on theuser's real time communication with one or more other users. The methodand system should be flexible and extensible to handle many userscenarios and usage patterns, yet straightforward for users toimplement.

SUMMARY OF THE INVENTION

Briefly, the present invention provides a system and method for improvedreal-time communications leveraging a personal computer and/or otherprocessing resources, in which the boundaries between various devicesand communication channels are blurred, as are the boundaries betweentypes of communication, including voice, text and video. The channel onwhich an incoming call is being received, as well as the device on whichthe call may be answered, are made significantly less important to theuser than the fact that a call is coming in from another party. Outgoingcalls focus on connecting to the called party, rather than anyparticular device.

When connected, the user may easily move between modalities, includingswitching devices on which the user is communicating, and transitioningbetween types of communication. For example, two or more users mayconnect with voice only communication, and then, with relatively littleeffort, one or more may move to voice and video. One or more users mayemploy instant messaging (IM) or similar text communication, such as toenhance an existing connection, or to avoid receiving a voice call butstill respond in some manner.

To provide such real-time communications functionalities, a computersystem includes various integrated communications-related devices, suchas an attached camera, an attached telephone handset, and an attachedspeaker and microphone, which may function as a speakerphone. Otherdevices include mobile communications devices such as a cellulartelephone or other mobile computing device. An auxiliary display devicesuch as for displaying caller ID data is coupled to the computer, butmay remain functional when the computer operating system is not running.Buttons related to controlling real-time communications, and indicatorsrelated to the state of real-time communications are also provided.

The present invention provides a real-time communications architectureincluding a controller comprising software that facilitatingcommunications in general, switches communications to and fromappropriate devices, and/or transitions communications among the variouscommunications channels and/or types of communications. For example, thereal-time communications controller couples the communication devices tovoice over IP, video, POTS, PBX, cellular networks and virtually anycommunications medium. The devices may be wired to the computer systemto which the real-time communications controller software is running, ormay be wireless, e.g., Bluetooth™ based devices. The user may interactwith the real-time communications controller through user interfacesoftware of one or more programs, and/or user interface software of thereal-time communications controller, but in one implementation, may alsodo so without running user interface software. To this end, thereal-time communication control buttons are easy to use and understand,and along with the indicators are positioned at locations thatintuitively facilitate interaction with the real-time communicationscontroller.

In general, the user is able to input data via any input mechanism,e.g., any microphone on any device, via any text or pointing-based userdata entry mechanism such as the keyboard and/or mouse, and/or thecamera. Similarly, the real-time communications controller is able toreceive data and output appropriate data to the user on any of theoutput mechanisms when activated, such as the speaker of a speakerphone,the handset telephone, a headset, the mobile phone or similar computingdevice, a PBX phone and/or a network card. Video, graphical and/or textinformation can also be displayed on the main display and/or theauxiliary display when available.

By responding to the user's actions, mode control logic in the real-timecommunications controller follows various real-time communication rulesand user-configured settings to handle a users communication needs, andin one implementation, can do so when the computer system is otherwisepowered down. The mode control logic also appropriately responds toexternal events, such as incoming calls, pages, text messages, forwardedvoicemail messages, and so forth. User actions may comprise lifting orreplacing the attached handset, typing and pointing-and-clicking a mouseor the like into a user interface, pressing phone controls buttons,issuing voice commands, and so forth.

The real-time communications controller also follows various real-timecommunication rules and user-configured settings to handle incomingcalls in a manner that matches a users communication needs. For example,for an incoming call, the mode control logic will display the caller IDinformation, sound a ring or call waiting tone on the speaker asappropriate, and respond to a user action such as selection of a deviceto answer the call by routing the voice, text and/or video asappropriate. Any other audio may be automatically muted until no callsremain.

The present invention allows a user to switch devices, transitionbetween communication channels, and transition between types of calls.Thus, for example, a user can answer a cellular phone call via theheadset, handset or speakerphone, add video to a call when available,switch a POTS call to a cell-phone carrier, and vice-versa.

Improved caller ID navigation is also provided, in which a user canswitch among parties on hold, and also conference in multiple parties,in a non-linear manner. A user can use any active display device such asthe auxiliary display to perform such operations.

Other advantages will become apparent from the following detaileddescription when taken in conjunction with the drawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram generally representing a computer system intowhich the present invention may be incorporated;

FIG. 2 is a general representation of a computer system arranged withcommunications-related mechanism in accordance with an aspect of thepresent invention;

FIG. 3 is a block diagram generally representing the central control ofcommunication devices and connections to communications channels, inaccordance with an aspect of the present invention;

FIG. 4 is a block diagram generally representing components to couplevarious devices and control and display connection state in accordancewith an aspect of the present invention;

FIG. 5 is a state diagram generally representing a dialer object andrelated operation in accordance with an aspect of the present invention;

FIG. 6 is a state diagram generally representing a ringer object andrelated operation in accordance with an aspect of the present invention;

FIG. 7 is a state diagram generally representing a line device objectand related operation in accordance with an aspect of the presentinvention;

FIG. 8 is a state diagram generally representing a phone device objectand related operation in accordance with an aspect of the presentinvention;

FIG. 9 is a general state diagram representing states of operation whenmaking a call, receiving a call, and when a call is waiting, inaccordance with an aspect of the present invention;

FIGS. 10-12 comprise a state diagram representing the transitionsbetween instant messaging, voice and video states in accordance with anaspect of the present invention;

FIGS. 13-15 are representations of an example display which allowsnon-linear selection of calls displayed based on caller ID information;and

FIGS. 16 and 17 are representations of an example display which allowsnon-linear conferencing of calls displayed based on caller IDinformation.

DETAILED DESCRIPTION Exemplary Operating Environment

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to: personal computers, server computers, hand-heldor laptop devices, tablet devices, multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputers, mainframe computers,distributed computing environments that include any of the above systemsor devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, and so forth, whichperform particular tasks or implement particular abstract data types.The invention may also be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in local and/or remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of the computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

The computer 110 typically includes a variety of computer-readablemedia. Computer-readable media can be any available media that can beaccessed by the computer 110 and includes both volatile and nonvolatilemedia, and removable and non-removable media. By way of example, and notlimitation, computer-readable media may comprise computer storage mediaand communication media. Computer storage media includes volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information such as computer-readableinstructions, data structures, program modules or other data. Computerstorage media includes, but is not limited to, RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical disk storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canaccessed by the computer 110. Communication media typically embodiescomputer-readable instructions, data structures, program modules orother data in a modulated data signal such as a carrier wave or othertransport mechanism and includes any information delivery media. Theterm “modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia includes wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, RF, infrared and otherwireless media. Combinations of the any of the above should also beincluded within the scope of computer-readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136 and program data 137.

The computer 110 may also include other removable/non-removable,volatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media, discussed aboveand illustrated in FIG. 1, provide storage of computer-readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146 and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers herein to illustrate that,at a minimum, they are different copies. A user may enter commands andinformation into the computer 110 through input devices such as atablet, or electronic digitizer, 164, a microphone 163, a keyboard 162and pointing device 161, commonly referred to as mouse, trackball ortouch pad. Other input devices not shown in FIG. 1 may include ajoystick, game pad, satellite dish, scanner, or the like. These andother input devices are often connected to the processing unit 120through a user input interface 160 that is coupled to the system bus,but may be connected by other interface and bus structures, such as aparallel port, game port or a universal serial bus (USB). A monitor 191or other type of display device is also connected to the system bus 121via an interface, such as a video interface 190. The monitor 191 mayalso be integrated with a touch-screen panel or the like. Note that themonitor and/or touch screen panel can be physically coupled to a housingin which the computing device 110 is incorporated, such as in atablet-type personal computer. In addition, computers such as thecomputing device 110 may also include other peripheral output devicessuch as speakers 195 and printer 196, which may be connected through anoutput peripheral interface 194 or the like.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a server, arouter, a network PC, a peer device or other common network node, andtypically includes many or all of the elements described above relativeto the computer 110, although only a memory storage device 181 has beenillustrated in FIG. 1. The logical connections depicted in FIG. 1include a local area network (LAN) 171 and a wide area network (WAN)173, but may also include other networks. Such networking environmentsare commonplace in offices, enterprise-wide computer networks, intranetsand the Internet. For example, in the present invention, the computersystem 110 may comprise source machine from which data is beingmigrated, and the remote computer 180 may comprise the destinationmachine. Note however that source and destination machines need not beconnected by a network or any other means, but instead, data may bemigrated via any media capable of being written by the source platformand read by the destination platform or platforms.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160 or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on memory device 181. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

Real Time Communications Modality Blurring

The present invention is generally directed towards blurring variouscommunications modalities, such that a user communicates with one ormore other users typically without having to deal with the connectionchannel or any particular device considerations. For example, audio,video and text communications will be initiated by the user using anysuitable device that the user has available and wants to use, regardlessof the channel (or channels) on which the communication ultimatelyoccurs. Similarly, received communications can be handled by anyappropriate device that the user selects. As will be understood, thevarious channels, devices and scenarios described herein are onlyexamples, and there are many others to which the present invention willapply.

Turning to FIG. 2 of the drawings, there is shown a computer system,such as the computer system 110 of FIG. 1, in which variouscommunications-related devices are integrated. For example, the computersystem 110 includes a monitor 191 having an attached camera 202, anattached telephone (e.g., handset) 204, and an attached speaker 206,which along with a microphone 208, may function as a speakerphone. Otherdevices not separately represented in FIG. 2 include mobilecommunications devices such as a cellular telephone or other mobilecomputing device capable of connecting in some way (e.g., by wiredcradle or Bluetooth™ technology) with the computer system. Althoughshown as individual devices in FIG. 2, it is understood that any or allof these devices, such as the speaker 206 and microphone 208, may becombined into a unitary assembly, or built into other devices, such asthe monitor 191. In any event, each of these devices may be present andif so are connected directly or indirectly by a wired or wirelessconnection to interfaces at the main PC housing 214 (containing at leastthe motherboard, but also typically containing some non-volatilestorage).

Also represented in FIG. 2 is an auxiliary display device 220, such asfor displaying caller ID data or like information as described below.The auxiliary display 220 is typically small (relative to the maindisplay screen 191), and may be the display of another device, such asthe screen of a mobile computing device, but may also be a dedicateddisplay, such as one that is operably connected or coupled to thecomputer operating system when the computer system 110 is in apowered-up state. The auxiliary display may comprise a simple textdisplay, such as a two-line display or the like. Further, note that theauxiliary display need not be an actual display, but can be a projection(e.g., onto a wall) of the information. As will become apparent below,the auxiliary display device 220 may provide some of its own processingand storage resources, so as to remain functional to some extent whenthe computer operating system is not running. In one alternativeimplementation described below, the auxiliary display device 220includes control buttons or the like thereon for interacting with thatdisplay 220, and possibly with the computer system 110 in general, asdescribed below. The keyboard 162 and/or monitor 191 (and/or possiblythe pointing device 161) may also include non-conventional buttons 222,such as buttons related to controlling real-time communications, andnon-conventional LED indicators 224, such as indicators related to thestate of real-time communications.

In accordance with an aspect of the present invention, as generallyrepresented in FIG. 3, the computer system 110 includes, a real-timecommunications controller 300, primarily implemented in a softwarecomponent or set of software components, that intelligently handlesvarious aspects of a user's real-time communication needs, includingfacilitating communications in general, switching communications to andfrom appropriate devices, and/or transitioning communications among thevarious communications channels and/or types of communication. Note thatthe real-time communications controller 300 need not be built into thecomputer system 110, but can connect thereto as an add-on device, suchas via a USB connection.

To handle a user's various communications needs, the real-timecommunications controller 300 allows a user to connect to variouscommunication channels (e.g., a telephone company switching network, amobile phone network, the internet or an intranet, and virtually anyother local area or wide area network over which communication ispossible) using the various communications-related data input/outputdevices. In general, the user is able to input data via any inputmechanism, e.g., any microphone on any device, via any text orpointing-based user data entry mechanism 302 such as the keyboard and/ormouse, and/or the camera 202, and have that data configured asappropriate (e.g., converted from speech to text) and sent to anappropriate (one or possibly more) of the available communicationchannels. Similarly, the real-time communications controller 300 is ableto receive data and output appropriate data to the user on any one ormore of the output mechanisms, such as the speaker of a speakerphone 304when the user has selected that device, the handset telephone 204 whenlifted, a headset 306 if activated, the mobile phone or similarcomputing device 308 if selected, a PBX phone 310 connected to a PBXmechanism 312 and/or a network card 314. When video, graphical and/ortext information is available, data can also be displayed on the maindisplay 191 (if active), and/or the auxiliary display 220. Note thatsome of the devices are represented in FIG. 3 as directly connected(coupled) to the computer system while other are shown as beingwireless, however it is understood that any device may be wired orwireless, e.g., a typical mobile computing device or mobile telephonemay be cradled and thus “wired” to the computer, and/or may integrate acellular phone, Wireless Ethernet (also known as 802.11b or Wi-Fi),and/or Bluetooth™ wireless technology. Further, any “connection”referred to herein and the like should be considered equivalent to a“coupling,” whether directly or indirectly connected through any type ofintermediary.

In accordance with an aspect of the present invention, the real-timecommunications controller 300 controls the communication modes for theuser, thereby removing much of the user's need to deal with device andconnection considerations. As generally represented in FIG. 4, this isaccomplished by responding to the user's actions, via mode control logic400 in the real-time communications controller 300 that follows variousreal-time communication rules 402 and/or is based at least in part onother user-configurable settings and a current operating state 404. Themode control logic 400 likewise responds to external events, such asincoming calls, pages, text messages, forwarded voicemail messages, andso forth.

User actions can occur and be detected in any number of ways, and thepresent invention contemplates essentially any way of providing useraction data or intended action data to a computer system. For example,such actions can include lifting or replacing the attached handset,which correspondingly actuates a hook switch 408 that is connected tothe real-time communications controller 300, and thus can be detected asstate data. Phone controls 410, such as in the form of buttons on adevice such as a talk button or buttons on the keyboard or monitor,(e.g., a speakerphone button, a flash button, and so on) can similarlyprovide user action information, as can a PBX telephone 310 or othercomputing device/telephone 308 (FIG. 3) connected to the computersystem. Controls 420 accompanying the auxiliary display 220, such as inthe form of scroll buttons and possibly an enter or select button (orequivalent button elsewhere on the system such as the flash button), canalso provide user action data, as can the computer data entry mechanisms302 such as a keyboard and mouse. Any microphone, such as the microphone208, can also provide user action data, e.g., via speech recognition(command and control) performed with a speech to text recognizer 430.Text may also be converted to speech by an appropriate converter 432,such as when the user enters text to leave a message, but insteadactually leaves a voice message on a called device, e.g., when the logic400 detects that forwarding text would not be appropriate. User actioninformation may also be communicated by wireless radio 434, and/or froma network connection or other communication channel. Motion and/orproximity sensing can likewise serve as a user action sensor.

Where appropriate, a graphical user interface component 440 shown aspart of the mode control logic 400 (but possibly anywhere in thecomputer system, such as in another application program) may facilitateentry of the user action information. For example, the user may interactwith one of the displays to select a displayed telephone number forconnecting or reconnecting with a person or device at that number.

The present invention similarly outputs information to the user in anyfeasible manner, including via the user interface when appropriate.Essentially, anything capable of outputting something that can be sensedby a user can serve as an output mechanism, including the displays 191and/or 220, lights and other indicators 224, the speaker 206 or otheraudio generating device such as a ringer on a cell phone, and othermechanisms such as one that provides tactile feedback (e.g., a vibratingcell phone). Thus, as used herein, the term “display” is only anexample, and is not limited to visible information, but can include anysensed output. For example, a hook indicator in the form of an LED maybe configured to flash red to indicate an incoming call, be lit greenwhen off hook, blink amber when the call is on hold, and blinkon-and-off when a call is muted. A tone and pattern can “display” thesame information audibly.

In keeping with the present invention, the related hook switch 408,controls 410, 420, displays 191 and 220 and indicators 224 provideadditional value that is consistent with but go beyond a user's pastcommunications experiences. To this end, the various controls andindicators are generally placed in proximity to the functions that theycontrol and represent. Further, the various devices such as handsets,headsets, and speakerphones are already familiar to users, and thephysical interfaces for these devices are largely equivalent toconventional devices.

Moreover, the present invention allows for various levels of computingskill. In general, the various control buttons 410, 420 and the hookswitch 408 that are provided allow the user to take advantage of much ofthe real-time communications controller's functionality without everhaving to operate the graphical user interface software 440. However, toaccomplish more sophisticated operations as desired, a user can alsoconnect via the user interface 440, e.g., to automatically dial calls,answer via a mouse click, customize settings and preferences and soforth.

In general, the logic 400 receives user action data or remotelycommunicated data (such as an incoming call), and based on variousreal-time communications rules 402 and user configurable settings 404,responds in a proper manner. For example, as described below, for anincoming call, the mode control logic 400 will display the caller IDinformation on the auxiliary display 220, or possibly the main display191 if so configured by the user, sound a ring or call waiting tone onthe speaker 206, and wait for a user action. If the user presses a phonecontrol button such as a speakerphone button, the call will be connectedto the appropriate device, e.g., routed to the speaker 206 andmicrophone 208. Any other audio, such as music that the user waslistening to, may be automatically muted until the call is complete,e.g., until the user again presses the speakerphone button todisconnect.

The present invention thus automatically and transparently responds inan appropriate manner to a user's needs. By way of example, consider theexample above of a user who is working at the computer when a call comesin. Instead of figuring out what type of call it is, finding the devicenecessary to (possibly) answer it, and so on, the user merely glances atthe auxiliary display, decides whether to take the call, and if so,selects any appropriate device to answer. Because the real-timecommunications controller 300 is the center of communication, a call toany of the user's landline telephone devices or any (smart) cellulartelephone will relay the caller ID data to the real-time communicationscontroller 300, which will display that caller ID data displayed on thesame auxiliary display 220 as any other call. For the same reason, theuser can answer any incoming call on the speakerphone 304, the handset204, the headset 306, the cellular telephone or mobile computing device308 and so on. The user need not even know whether a given call came inon a given communications channel, such as the cellular channel insteadof a POTS channel or network channel, although for reasons such as costof airtime, the user may want to know this information. Regardless ofthe device and the channel, the user is provided access to a universaladdress book, call log, and inbox.

Moreover, the present invention allows a user to transition betweencommunication channels. Thus, for example, a user who receives a POTStelephone call can answer talk for awhile, and switch that call over toa mobile phone to continue the conversation away from the computersystem. Note that this is not conventional call forwarding set up inadvance for switching at the central switching office or the like, butrather on-demand transitioning during a telephone call. For example,rules 402 can be set up to favor landline over cellular for local callssince there is no airtime charge, however long distance may be favoredover cellular when there is no additional charge and the user is likelyto have unused airtime before the next billing cycle. Voice over IP maybe the most favored, but may not be available when the computer systemis powered down.

One simple way to accomplish transitioning live connections is via adifferent telephone line relative to the source of the call. Forexample, if a user is leaving his or her office, the real-timecommunications controller 300 can transparently (or mostlytransparently) establish a cellular connection to the user's mobiletelephone device on that other line and connect the other caller to thatmobile telephone device through the real-time communications controller300. Voice over IP can similarly be used.

Three-way and conference calling capabilities can also be leveraged,e.g., a user can walk into his or her office while connected to anotherperson via a cell phone, the real-time communications controller 400 canestablish another call to the other party via a POTS, PBX or voice overIP call, and when connected, terminate the cellular phone call.

In addition to switching and transitioning among devices and channels,the present invention can switch and/or transition among types ofcommunication. For example, a user can respond to a telephone call withan instant message, either automatically or by typing/speechrecognition, e.g., to notify the caller that the call was received butcould not be answered at that moment. In the event that the real-timecommunications controller 300 has the caller ID and/or other informationfrom that call, possibly including preset knowledge such as that theincoming telephone number belongs to a mobile device, the real-timecommunications controller 300 can adjust the instant message asappropriate, e.g., convert a text message to speech to leave a voicemessage when the call was known to be from a mobile phone that does nothandle text, or when the user is known to be traveling. Note that theinstant message does not have to be sent to the same device that calledthe user, e.g., a user can respond to a received phone call from aconventional telephone with an instant message to the caller's computersystem. A simple database lookup or the like by the mode control logiccan provide the recipient's address from the caller ID data so that theuser does not even have to enter this information.

As another type change, when available, the present invention can startwith an audio call and add video (one-way or two-way) and/or instantmessaging as desired by the parties, or start with an instant messageand add audio and/or video. For example, a user can press a camerabutton, or select video via a mouse click, to establish a videoconnection and start transmitting video. A video call can similarly betransitioned to a non-video (e.g., audio-only) call. In general, thepresent invention facilitates switching and/or transitioning between anymodes of communication. Transitioning between text, voice and videostates is described below with reference to FIGS. 10-12.

It should be noted that the computer system need not be fullyoperational for communications to work in accordance with the presentinvention. Indeed, communications should still work when the computer ispowered down, at least to a default extent or to an extent configured bya user, such as when the computer system is in a sleep state or ahibernate mode, and/or when the user is locked out of the system viasecurity mechanisms. Note that the type and extent of communications maybe configured by the user. For example, the user may want the telephonehandset, speakerphone and caller-ID device to work as conventionalappliances when the computer system is powered down, but not the camera,headset, or network card. Further, the user may limit the extent of thecommunications based on the computer system state, e.g., when the useris not logged in, the real-time communications controller 300 may be setto block long distance calls, block calls to a certain prefix, and soforth.

To enable and control communications in these powered down modes, thereal-time communications controller 300 may be loaded into executablenon-volatile memory, operated with a secondary processor, and so forth,so that communications works as long as some power is available, eventhough the disk, main processor, main display, network card and/or otherparts of the system are powered down.

Turning to an explanation of the operation of the present invention withparticular reference to the mode control logic 400, FIGS. 5-8 representgeneral logic and rules in the form of state diagrams for variousobjects that control behavior of the various mechanisms. For example,FIG. 5 shows the logic and rules for connecting a call when initiatingthe call via a dialer object or the like. As represented in FIG. 5, whenstarting in an idle state 500, the user will send a dial request to linedevice (FIG. 7). In the event that the computer system is notpowered-down, a dialer program may be automatically displayed on thegraphical user interface (GUI) on the main display 191, from which theuser may enter a state where the user can select a person to call orclose the dialer, or the user may dial (or hang-up) in a conventionalfashion. A user may also be able to scroll through and select a numberon the auxiliary display 220, such as to call a number from a list ofcalls logged via caller ID information, speak a name or number into amicrophone, press a speed dial button and so on. If dialed in anymanner, the logic will attempt to connect the call, and if successful,the system will end up in a connected state 504 until terminated.

FIG. 5 also shows that a call answered (in any manner) when in the idlestate 500 goes directly to the connected state 504. More particularly, apick-up request is sent to the line device (FIG. 7). If no receiver isactive/off hook, a default (which may be user-configurable) receiver isselected as the active receiver, and the audio stream is sent to thatactive receiver. The call event may be logged in the system.

When connected, if the connection supports video and a video camera ispresent and its use is allowed, video is streamed. A user may disallowvideo by closing the camera's shutter, or by configuring the system tonot automatically allow video, that is, only allow video if some manualuser action is detected. The user can see when an incoming call supportsvideo and can answer the call immediately in that mode, such as bypressing a camera button. Optionally, the user can set a preference sothat the camera is automatically enabled whenever answering a call thatsupports video.

If the connection supports text or rich ink messaging, an embeddedinstant messenger (IM) chat control is also enabled. If an indicatorserves as a “connection” light, i.e., to indicate when a communicationlink is established between two users, then the indicator may alsoindicate when an IM connection is active, such as by a different LEDbehavior (e.g., color and/or flash pattern) with respect to a voice-onlycall, to communicate to the user that an IM connection exists.

FIG. 6 shows other operation of the logic 400, such as with respect toan object that controls a ringer or other suitable notificationmechanism when an incoming call is received. When in a ringer idle state600, (which means idle with respect to notifying a user of a newincoming call, and may include a state in which a user is already on acurrent call), the logic 400 will look up and display the new callerinformation and enter a waiting state 602. For example, the waiting callmay be displayed via a side bar notification fly-out if the main displayis on, and/or may be displayed on the auxiliary display device. At thattime, the logic 400 also may sound an audible alert (and/or providinganother suitable notification, such as flashing an LED and/or vibratinga device). The type of alert may depend on whether the user is onanother call or on other state information, e.g., if in a Do Not Disturbmode, no sound is played, else if no receiver is active, a ringing soundis played, otherwise a receiver call waiting tone is played.

If the active receiver goes off hook or is clicked, a flash button orthe like is pressed, or the user requests that the call be answered viathe graphical user interface, the ringer leaves the waiting state 602when the call is connected, after placing any current call on hold. Aconference button or the like (not shown) can add the caller to anexisting call. If no action is taken to connect the call, a voice mailtime out will send the call to voice mail and return the ringer to theidle state with respect to further incoming calls.

FIG. 7 shows the operation of the line device object in an idle state700, a waiting state 702, a connecting state 704 and a connected state706. In general, these states have been previously described above inslightly different context, and will thus not be repeated herein exceptto note that while attempting to establish a connection as representedby state 704, a connecting status or the like may be displayed by theline device object, during which time the user can cancel the attempt.Also, a hold request can transition the user from the connected state706 to a waiting state 702 by placing the line on hold, while a pick-uprequest works in the opposite way, by connecting the line. Also, notshown in FIG. 7, multimedia playback may be turned off or muted whileany receiver is active, and restored when no receiver is active. FIG. 8represents the operation of a phone device object in an idle state 800,and in two off hook states, a normal state 802 and a muted state 804.The transitions between the states are readily apparent in FIG. 8,although it should be noted that as described above, another receivercan be set as the active receiver without disconnecting the line. Forexample, selecting a speakerphone logically hangs-up an active handsetor headset, picking up the handset turns off the speakerphone orheadset, pressing a talk button on the headset turns off thespeakerphone or handset, and so forth.

FIG. 9 represents a general overview of the logic 400 that summarizesvarious states of operation. For example, as represented in FIG. 9, thehook switch operates in a number of ways, including using it toterminate a call, lifting it to make a receiver active and pick upwaiting line, as a flash button to switch lines or terminate a call ifno lines are waiting. The hook switch may be implemented as a cradlehook switch mechanically integrated into the handset cradle, comprisinga two-position toggle switch that closes when the handset is off hookand opens when the handset is in the cradle.

In general, as represented in FIG. 9 lifting the handset from the cradleusually takes the handset off hook and displays a dialog box forselecting the person to call. During a call, replacing the handset inthe cradle will hang up the call. If the handset is not the activereceiver during a call, lifting the handset from the cradle will makethe handset the active receiver, switching off the old active receiver(e.g., the speakerphone or headset). When the phone is ringing from anincoming call, lifting the handset will answer the call, taking thehandset off hook. If the phone is ringing while another phoneconversation is already in progress, depressing the cradle hook switchquickly will toggle between the two calls, like flashing the hook switchwill do with call waiting present on a legacy phone system. A talkbutton, usually integrated into the keypad of telephone devices, such asa headset or handset, performs similar functionality to the hook switchand can generally be considered similar to the hook switch operation inFIG. 9. In general, either a talk button or cradle hook switch (or atleast a graphical user interface equivalent) is required for each phonehandset or headset. The talk button comprises a single-throw, momentaryswitch that toggles the hook state for a telephone receiver. Its primaryfunction is to initiate or answer a call, and then hang up when theconversation is finished. During a call, pressing a talk button on theactive receiver will hang up the call. Pressing the talk button onanother receiver will have the effect of making it the active receiver,switching off the old active receiver (handset or headset). When thephone is ringing from an incoming call, pressing the talk button willanswer the call, taking the receiver off hook. If the phone is ringingwhile another phone conversation is already in progress, pressing thetalk button will toggle between the two calls.

FIG. 9 also shows various operations via a speakerphone button similarlycomprising a single-throw, momentary switch to toggle the hook state fora speakerphone. Its primary function is to initiate or answer a call,and then hang up when the conversation is finished. The speakerphonebutton is a special case of the talk button, used where the incomingcall is broadcast over a speaker and audio is collected via a far fieldmicrophone. In a cordless handset with speaker phone capabilities, itmay be important to indicate to the user that the speakerphone is activeto differentiate it from the handset talk switch.

If a virtual speakerphone is implemented using the microphone andspeakers that are already part of the PC system, a speakerphone buttonmay be located on the keyboard, e.g., to the left of the main keyboardkeys. This button usually takes the PC speakerphone off hook anddisplays a dialog box for selecting the person to call. During a call,pressing the speakerphone button when the speakerphone is the activereceiver will hang up the call. If the speakerphone is not the activereceiver, this will have the effect of making it the active receiver,switching off the old active receiver. When the phone is ringing from anincoming call, pressing the speakerphone button will answer the call,taking the speakerphone off hook. If the phone is ringing while anotherphone conversation is already in progress, pressing the speaker phonebutton will toggle between the two calls like flashing the hook switch.

Because the talk button or speakerphone button state may not be readilyapparent to a user, a hook indicator may be used in conjunction with atalk or speakerphone button, and, for example, may be located adjacentor integrated into the talk button or speakerphone button. The hookindicator may indicate state via various colors and flash patterns,e.g., steady state unlit when the receiver is on hook, steady stategreen when the receiver is off hook, or steady state red when no serviceis available. A flashing indicator, such as with a “wink” duty cycle canoverlay the steady state, e.g., a red wink overlay can indicate callwaiting, or an amber flash overlay can indicate a call holding. The callwaiting overlay indicator will continue to blink as long as the user canpick up the phone to connect to the incoming caller, even if the call isno longer ringing. This allows call screening when callers are recordingvoice mail. The call holding overlay is generally only needed when adedicated hold indicator is not present.

Although not specifically represented in FIG. 9, other buttons may beprovided. For example, a flash button, often used for alternate lineselection, generates a momentary on hook condition for signalingpurposes. When the user is notified of a call waiting, pressing theflash button will place the current call on hold and answer the incomingcall. With calls on hold, pressing the flash button places the currentcall on hold and picks up the next call in the hold queue. A hold buttonplaces a current call on hold by placing the current call into a callwaiting state as generally represented in FIG. 7, freeing the receiverto make another call, answer an incoming call, or hang up to pick up onanother receiver. A conference button initiates a conference call, inwhich pressing the conference button joins the current call with one ormore calls waiting on hold. Other buttons are feasible, such as a buttonto immediately send an incoming call to voice mail. Note that becausethe real time communications controller 300 handles the calls,conference calls can also be handled via the graphical user interface440, e.g., additional contacts or groups of contacts can be clicked anddragged to a window to conference in others.

FIG. 9 also represents a number of user interaction scenarios withrespect to real-time communications, including handling an incoming calland making a call. As described above, to make a phone call, the userlifts the handset or presses the talk switch on the speakerphone orheadset, and then dials a number. Alternatively, the user may select acontact from the display 220 or graphical user interface 440, or employvoice dialing. When initiating a call from the graphical user interface440, such as within the context of a communications application such asMicrosoft® Outlook, the user can initiate a phone call from any standardcontact object supported by an API. The operating system takes thedefault receiver off hook and makes the connection using a preferred orspecified telecommunications service.

To answer a phone call, the user lifts the handset, presses aspeakerphone switch, or presses a talk switch headset (or on thespeakerphone). Before answering, the user has the ability to view callerID, and possibly other pertinent contact information displayed with thenotification message (on the main display or auxiliary display),allowing the user to pre-screen the call. Depending on preferences, theuser can then choose to pick up the call or transfer it to voice mail,which may include transferring it to one of a plurality of custom orstandard voice mail messages. While the caller is recording voice mail,the user can monitor the incoming recording and pick up the receiver,similar to a traditional answering machine.

The examples below provide additional interaction detail for commontelephony functions, including an incoming call scenario, wherein thereal-time communications controller sounds the ringer and blinks thetalk button indicator, while using caller ID information to displaycontact information. When the user presses the talk button, a microphonebutton, camera button, or picks up the handset, or presses the hook(talk) switch on the headset, the real-time communications controller300 may light an indicator for the active receiver. The user receivesthe call, and can switch among devices, e.g., to switch to thespeakerphone, the user presses the talk button or mic button; to switchto the video speakerphone, the user presses the camera button; to switchto the handset, the user picks it up (activating the hook switch); or toswitch to the headset, the user presses the hook (talk) switch on theheadset. A cell phone or mobile computing device may be similarly used,even without actually switching the call to the cellular network (whichmay require some additional user action). When the user switches activereceivers, the system disables the previous receiver to preventfeedback, and updates indicators to reflect the active receiver.

If the call supports video, the real-time communications controllerblinks the camera indicator. If the user presses the blinking camerabutton, or automatic video is set by user preference to automaticallystream video when video is available, the system opens the videomonitor, lights the camera indicator, and starts streaming video.

To mute a component, the user can take an appropriate action. Forexample, to mute video and audio, the user presses the camera button,whereby the real-time communications controller blinks the audio andvideo indicators until the camera button is pressed again or the call isterminated. To mute video only, the user shuts the camera's privacyshutter, whereby the system un-lights the video indicators and closesthe A/V monitor, but maintains the audio indicator. To mute speakerphoneaudio, the user presses the microphone (speakerphone) button, wherebythe real-time communications controller blinks an appropriate indicatoruntil the button is pressed again or the call is terminated. To hang up,the user presses the hook switch for the active receiver.

To place a call, the real-time communications controller 300 reacts whenthe user presses the talk button, picks up the handset, or presses thehook switch on headset, by displaying a dialog box for selecting theperson to call. The system may also audibly ask through the activereceiver, such as when voice dialing is enabled, as to who the userwould like to call. The real-time communications controller usesresulting input to select a contact from the dialog box, and the usercan query to search and can re-query to refine the search, if necessary.

When the user makes a selection or acknowledges a match, the real-timecommunications controller shows contact information for the selectedrecipient and shows a status dialog box while the call is connecting.When connected, the user can switch devices as described above, and alsoswitch modes, e.g., audio to video, and so forth, with the indicatorsoperating in the above-described manner.

The user can also perform other communications-related tasks, includingchecking voice mail and call screening. For example, an indicator lightmay blink to indicate a voice mail, whereby the user can user press anotification button causing the real-time communications controller toenter a callback routine for the highest priority notification. In caseof voice mail or priority email, this may open the user's unified InBox. Call screening logic follows various rules, including showingcaller ID information and contact summary data until, after a presetnumber of rings, the system stops the ringer and sends a voice mailmessage to an incoming caller. If the call screening option is enabled,the system echoes the incoming message to a system audio output device(usually the speaker). If the user presses the hook switch of anyreceiver, the system discards the message and switches to real-timevoice, otherwise, when the user hangs up or after a certain time limit,the system stops blinking the talk button indicator and files a voicemessage.

FIG. 10-12 generally show the relationships between instant messaging,and/or voice communication states, with various incoming and/or outgoingvideo states, and the various channel switching transitions betweenthese states. Note that although instant messaging is used as an examplein FIG. 10, any other type of real-time text messaging is equivalent,including text messages sent to pagers, and possibly electronic mail ifrelatively fast enough to be considered real-time. One state representedin FIG. 10 is a no call state 1000 than can transition to various voiceand/or instant messaging states. As also represented in FIGS. 10-12, theconnected states can have automatic video active, or video can be added(or ended) at any time by either party.

FIG. 10 represents the instant messaging states without voice, includinga state with no video 1002, and states with incoming, outgoing andtwo-way video, states 1004, 1006 and 1008, respectively. As is readilyapparent, each user controls these states by user actions or configuredpreferences for automatic video operation. Note that voice can be addedfrom these states, in which event a new state will be entered asrepresented in FIG. 12.

FIG. 11 represents the voice without instant messaging states, withsimilar video-related states of voice only, incoming, outgoing andtwo-way video, states 1004, 1006 and 1008, respectively. Note thatinstant messaging can be added from these voice states, in which event anew state will be entered as represented in FIG. 12.

FIG. 12 represents the states and transitions when both instantmessaging and voice are active, with transitions within the same fourpossible video-related states of none, incoming, outgoing and two-wayvideo available. As can be seen in FIG. 12, the communication maytransition to a no instant messaging state (FIG. 11) or no voice state(FIG. 10), or no communication at all at state 1000 of FIG. 10, based onstraightforward actions of the users who are communicating.

In accordance with another aspect of the present invention, there isprovided a caller ID navigation mechanism that allows users to navigatea stack of calls in a non-linear manner. As represented in FIG. 13, theavailable calls may be displayed on the auxiliary display device 220,with the currently connected call 1302 indicated in some manner todifferentiate it from the calls on hold. As can be appreciated, richerinformation may be provided depending on the amount of display areaavailable, such as the length of time the user was on hold, the type ofconnections that are available and its current state (voice, video,instant message) and so forth.

In general, the user presses the scroll buttons 1304 and 1306 tonavigate to a call, such as represented by the dashed box 1410 in FIG.14, and presses a select button 1308 to switch to the call, asrepresented by the connected box 1502 moving relative to its position inFIG. 13. Because the real-time controller 300, and not a remote switchat the telephone is in charge, the switching may be non-linear. Notethat the scroll buttons are typically near or part of the auxiliarydisplay unit because the user will be looking at the display, howeverthe select button 1308 may be another button located elsewhere, such asa flash button on the keyboard, (as long as that button remains activewhile the computer system is otherwise in a powered off state).

Further, if the main display is active, the user need not scroll, butcan instead simply select a call to switch two via a point and clickoperation. Of course, the user can also scroll, such as via keyboardcursor keys to highlight, and the Enter key to select. Again, richerinformation may be provided, particularly since the main display 191will provide more display area relative to the auxiliary display 220.

As represented in FIGS. 16 and 17, (which shows a richer set ofinformation), depending on the available buttons and their meaning,conferencing by selecting users is also feasible. For example, holdingthe select button while on a connected call, and then scrolling throughthe list view, can keep any existing connections connected, andreleasing the select button can add (or drop if already selected)another user. On the main display, more elaborate schemes are available,e.g., to drag and drop other parties into a “connected” window, rightclick on a list view similar to that shown in FIGS. 16 and 17, and soforth.

As can be seen from the foregoing detailed description, there isprovided a real-time communications method and system for handling theswitching and transitioning between communications modalities includingswitching devices, switching communication channels and/or transitioningtypes of communications, e.g., audio, video and text. The presentinvention facilitates user communication by blurring the distinctionbetween these various modalities, allowing the user to focus on thecommunication rather than the mechanisms for accomplishing thecommunication. The method and system thus provide significant advantagesand benefits needed in contemporary computing and communications.

While the invention is susceptible to various modifications andalternative constructions, certain illustrated embodiments thereof areshown in the drawings and have been described above in detail. It shouldbe understood, however, that there is no intention to limit theinvention to the specific forms disclosed, but on the contrary, theintention is to cover all modifications, alternative constructions, andequivalents falling within the spirit and scope of the invention.

1. In a computing environment, a method comprising: receiving a call onone of a plurality of communications networks; providing a notificationof the call; and coupling the call to a selected communication deviceindependent of the communication network on which the call was received.